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GAO 


United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


Accounting  and  Information 
Management  Division 

B-259277 

December  9, 1994 

The  Honorable  Ronald  V.  DeUums 
Chairman,  Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  John  P.  Murtha 
Chairman,  Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 

The  Honorable  Sam  Nunn 

Chairman,  Committee  on  Armed  Services 

United  States  Senate 

The  Honorable  Daniel  K.  Inouye 
Chairman,  Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

As  directed  by  the  House  Committee  on  Armed  Services  in  its  report 
accompanying  the  National  Defense  Authorization  Act  for  fiscal  year  1995, 
we  reviewed  possible  civilian  agency  use  of  the  supercomputing 
capabilities  at  the  Department  of  Defense’s  (dod)  National  Test  Facility 
(ntf).  We  were  specifically  directed  to  include,  but  not  limit  our  study  to, 
using  these  resources  to  support  (1)  the  Department  of  Energy  in 
monitoring  oil  pipeline  leakage,  (2)  the  Department  of  Education  in 
developing  on-line  educational  software,  (3)  the  Department  of  Health  and 
Human  Services  (hhs)  in  developing  an  on-line  medical  imagery 
repository,  (4)  the  Federal  Emergency  Management  Administration  (fema) 
in  developing  innovative  emergency  reaction  simulations,  and  (5)  the 
National  Oceanic  and  Atmospheric  Administration  (noaa)  in  climatological 
modeling. 


Results  in  Brief 


Based  on  our  interviews  with  the  civQian  agencies  cited  in  the  House 
report,  none  would  be  able  to  make  effective  use  of  ntf’s  excess 
supercomputing  capabilities.  These  agencies  stated  they  could  not  use  the 
resources  primarily  because  (1)  ntf’s  supercomputers  are  older  machines 
whose  performance  and  costs  cannot  match  those  of  more  advanced 
computers  available  from  other  sources  and  (2)  some  agencies  have  not 
yet  developed  applications  requiring  supercomputer  capabilities  or  do  not 


Page  1  GAO/AIMD-95-28  NTF  Supercomputers 


B-259277 


Background 


have  funding  to  support  such  activities.  In  addition,  future  support  for  the 
hardware  and  software  at  ntf  is  uncertain,  making  any  investment  by  an 
outside  user  risky. 


NTF  was  designed  and  constructed  in  the  1980s  to  support  the  Strategic 
Defense  Initiative  Organization  (smo)  as  the  centerpiece  for  its  National 
Test  Bed,  a  network  of  research  facilities  linked  together  to  test  and 
simulate  software  and,  to  a  lesser  extent,  hardware  that  might  be  used  in 
defending  the  United  States  against  nuclear  baUistic  missile  attack,  smo 
was  later  reorganized  as  the  Ballistic  Missile  Defense  Organization  (bmdo). 

Opened  in  1990,  ntf  is  a  $123  million  government-owned, 
contractor-operated  facihty  that  features  a  sophisticated,  reconfigurable 
design  and  extensive  network  connectivity  that  give  it  the  potential  for 
conducting  large-scale  computer  operations.  Located  near  Colorado 
Springs,  Colorado,  ntf  is  also  a  uniquely  secure  facility  that  meets  dod’s 
top  security  requirements. 

Although  NTF  was  originally  built  to  house  21  Cray-2  supercomputers  to 
support  smo  activities,  only  2  Cray-2s  were  acquired  and  installed.  The 
larger  supercomputer  features  four  central  processors  and  612  million 
words  of  memory.  It  is  currently  running  at  capacity  at  the  secure 
classified  level,  almost  totally  in  support  of  bmdo  needs.  A  smaller  system, 
featuring  two  central  processors  and  128  miUion  words  of  memory,  was 
turned  off  last  year  to  save  operational  and  maintenance  expenses.  When 
this  machine  is  operating,  its  maintenance  costs  up  to  $70,000  monthly. 

Changes  in  the  world  geopolitical  situation  and  the  diminished  threat  of  a 
deliberate  Soviet  nuclear  attack  have  led  to  changes  in  ntf’s  mission  and 
funding  level.  The  volume  of  work  associated  with  supporting  simulation 
and  testing  of  missile  defense  hardware  and  software  has  diminished. 

BMDO  funding,  which  accounted  for  89  percent  of  ntf’s  budget  in  fiscal  year 
1994,  has  fallen  from  $140  million  in  fiscal  year  1993  to  $76.5  miUion  in 
fiscal  year  1994.  Fiscal  year  1995  funding  remains  uncertain.  Based  on  the 
President’s  budget,  ntf  anticipates  receiving  $62  million  in  bmdo  funding 
this  year. 

Faced  with  such  reductions,  ntf  officials  have  begun  looking  for  external 
users  to  help  cover  operational  costs.  Varied  services  have  been  offered, 
ranging  from  simply  leasing  secure  floor  space  at  ntf  to  contracting  for 
complete  software  development  and  implementation  projects,  ntf  officials 
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have  also  proposed  restarting  the  smaller  Cray-2  system  and  selling 
processing  time  on  it  to  outside  users.  Policies  and  procedures  for  this 
service  have  been  developed,  but  no  customers  have  yet  been  found. 

Scope  and 
Methodology 

■ 

To  identify  possible  civilian  agency  use  of  ntf’s  supercomputing 
capabUities,  we  spoke  with  responsible  program  officials  at  the  Ballistic 
Missile  Defense  Organization  and  visited  ntf.  At  ntf  we  assessed  the 
supercomputer  capabilities  of  the  facility  and  determined  what  excess 
capabUities  are  avaUable  to  outside  users  and  the  costs  for  using  these 
capabilities.  We  then  interviewed  scientific  researchers  and  officials  at 
government  laboratories  about  these  capabilities.  Finally,  we  spoke  with 
officials  from  the  relevant  programs  at  the  specific  civilian  agencies  cited 
in  the  act  to  ascertain  their  interest  in  using  ntf’s  supercomputer 
resources.  The  results  of  our  discussions  with  these  agency  officials  are 
summarized  in  appendix  I. 

We  conducted  our  review  between  August  1994  and  October  1994,  in 
accordance  with  generally  accepted  government  auditing  standards.  As 
requested,  we  did  not  obtain  dod’s  comments  on  this  letter.  However,  we 
provided  a  draft  of  this  letter  to  ntf’s  Assistant  Deputy  for  Engineering, 
who  generally  agreed  with  the  facts  as  presented.  We  have  incorporated 
the  Assistant  Deputy’s  views  as  appropriate. 

The  Civilian 
Community  Cannot 
Use  Excess  NTF 
Capabilities 

■ 

Officials  firom  civilian  agencies  and  laboratories  we  spoke  with  cited 
several  reasons  they  would  be  imable  to  use  ntf’s  excess  supercomputing 
resources.  First,  ntf’s  supercomputers  are  older  machines  whose 
performance  and  costs  cannot  match  those  of  more  advanced  computers 
available  firom  other  sources.  Second,  some  agencies  have  not  yet 
developed  Explications  requiring  supercomputer  capabilities  or  do  not 
have  funding  to  support  such  activities.  'Third,  because  future  support  for 
NTF  hardware  and  software  is  uncertain,  any  investment  by  an  outside  user 
could  be  risky. 

NTF’s  Older 
Supercomputers  Are  No 
Longer  Competitive 

Acquired  in  the  late  1980s,  the  Cray-2  systems  installed  at  ntf  no  longer 
represent  state-of-the-art  supercomputer  technology.  For  example,  the 
latest  Cray  supercomputer,  the  Y-MP  C90  system,  equipped  with 

16  processors  and  256  million  words  of  memory,  can  perform  at  speeds  5 
times  faster  than  the  Cray-2.  Moreover,  the  latest  generation  of 
workstations,  such  as  the  IBM  RS/6000  690  or  the  Silicon  Graphics  TFP, 
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are  far  more  economical  and  can  almost  match  the  sustained  performance 
of  one  of  the  Cray-2’s  processors. 


Because  of  the  Cray-2’s  processing  speed,  the  $160  hourly  rate  ntf 
proposed  for  using  the  system  is  not  competitive  with  rates  charged  by 
other  centers  for  newer,  faster  supercomputers.  In  fact,  when  the  rates  are 
expressed  in  standard  terms  (that  is,  price  per  million  instructions  per 
second),  the  proposed  ntf  Cray-2  rate  turns  out  to  be  twice  the  price 
several  of  the  National  Science  Foundation  sponsored  supercomputing 
centers — such  as  the  San  Diego  Supercomputing  Center  or  the  National 
Center  for  Atmospheric  Research— charge  for  time  on  their  newer  and 
faster  Crays. 


Not  all  the  agencies  we  contacted  have  the  ability  to  use  ntf 
supercomputer  resources,  regardless  of  the  desirability  of  those  resources. 
Agencies  such  as  the  Department  of  Education  and  fema  do  not  have 
budgets  for  developing  apphcations  to  take  advantage  of  the  power  of  a 
supercomputer.  Of  all  the  agencies  we  visited,  we  identified  only  one 
project,  at  hhs,  that  could  potentially  need  supercomputer  time.  However, 
this  project,  the  National  Library  of  Medicine’s  Visible  Human  Project,  will 
not  need  access  to  a  supercomputer  before  1996. 


Cray  Research,  Inc.,  stopped  manufacturing  the  Cray-2  series 
supercomputer  in  1989.  Although  the  company  currently  supports  and 
maintains  existing  Cray-2  systems,  there  is  no  assurance  that  it  wfil 
continue  to  do  so.  Cray  has  already  informed  its  customers  that  after 
September  1996  no  new  software  upgrades  wfil  be  provided.  In  addition, 
hardware  maintenance  is  likely  to  be  difficult  or  more  expensive  to  obtain 
after  that  time. 

Future  support  for  ntf’s  supercomputer  resources  is  further  jeopardized 
by  the  uncertainty  of  funding  to  replace  the  Cray-2s  and  to  retain  the 
present  cadre  of  talented  support  staff.  The  xmcertainty  surrounding  future 
support  and  maintenance  for  ntf  supercomputers  increases  the  risks  for 
new  users  in  buying  supercomputer  time  from  ntf  because  they  might  not 
realize  a  full  return  on  the  initial  investment. 

We  are  sending  copies  of  this  letter  to  the  Ranking  Minority  Members  of 
the  House  and  Senate  Committees  on  Appropriations  and  the  House  and 


Future  Support  and 
Maintenance  for  NTF 
Supercomputers  Is 
Uncertain 


Supercomputer  Funding 
and  Applicability  Are 
Unclear  for  Some  Agencies 
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Senate  Committees  on  Armed  Services,  the  Secretary  of  Defense,  the 
Secretary  of  the  Air  Force,  the  Director  of  bmdo,  and  other  interested 
parties.  Copies  will  also  be  made  available  to  others  upon  request. 

If  you  or  your  staffs  have  questions  or  wish  to  discuss  the  issues  in  this 
report  further,  please  contact  me  at  (202)  512-6240  or  John  de  Ferrari, 
Assistant  Director,  at  (202)  512-6335.  Mqjor  contributors  to  this  report  are 
listed  in  appendix  II. 


■^Jack  L.  Brock,  Jr. 

Director,  Information  Resource 
Management/National  Security  and 
International  Affairs 
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DOE  Department  of  Energy 

FEMA  Federal  Emergency  Management  Administration 
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Appendix  I 

Potential  Civilian  Agency  Use  of  Excess 

NTF  Supercomputing  Resources 

Support  to  the 
Department  of  Energy 
in  Monitoring  Oil 
Pipeline  Leakage 

The  Department  of  Energy  (doe)  does  not  use  supercomputers  to  monitor 
oil  pipeline  leakage.  Only  one  program  in  which  doe  participates  within 
the  Oil  and  Gas  Exploration  and  Production  Division  uses 
supercomputers— the  Natural  Gas  and  Technology  Partnership.  However, 
this  program  is  not  interested  in  ntf’s  supercomputing  resources  because 
program  officials  do  not  consider  the  Cray-2  powerful  enough  for  their 
global  climate  monitoring  simulations,  which  they  currently  run  on 
advanced  parallel  processing  machines. 

Support  to  the 
Department  of 
Education  in 
Developing  On-line 
Educational  Software 

The  Department  of  Education  has  no  funds  to  develop  educational 
software  to  run  on  supercomputers.  Moreover,  the  Department  is  just 
beginning  to  research  how  supercomputers  could  be  used  as  educational 
tools  and  has  not  yet  determined  what  directions  are  worthwhile  pursuing. 
Since  the  National  Science  Foundation  funds  activities  to  develop 
educational  applications  for  supercomputers,  the  user  community  looks  to 
them,  rather  than  the  Department  of  Education,  for  support. 

Support  to  the 
Department  of  Health 
and  Human  Services 
in  Developing  an 
On-line  Medical 
Imagery  Repository 

The  National  Library  of  Medicine’s  Visible  Human  Project  is  the  only  hhs 
project  developing  a  large  repository  of  medical  imagery.  However,  project 
staff  are  just  now  acquiring  their  first  data,  which  are  being  stored  on  a 
conventional  Unix  workstation.  Supercomputer  apphcations  for  this 
project  will  not  be  needed  until  sometime  in  1996  when  the  repository  is 
scheduled  to  come  on-line  and  hhs  is  to  begin  developing  algorithms  for 
researchers  to  view  and  manipulate  the  data  At  that  time,  hhs  officials 
said  they  will  first  consider  using  supercomputing  resources  already 
available  at  the  National  Institutes  of  Health.  If  these  prove  unavailable, 
they  might  be  interested  in  using  ntf’s  resources.  As  noted  above, 
however,  hardware  and  software  support  to  the  Cray-2  at  ntf  will  be 
uncertain  after  1996. 

Support  to  FEMA  in 
Developing  Innovative 
Emergency  Reaction 
Simulation 

FEMA  has  neither  the  funds  nor  the  expertise  to  develop  emergency 
reaction  simulation.  They  currently  contract  with  the  Defense  Nuclear 
Agency  to  develop  disaster  models,  which  fema  then  runs  in-house  on 
high-end  workstations.  The  models  run  effectively  on  these  workstations 
and  do  not  require  a  supercomputer. 

Support  to  NOAA  in 
Climatological 

Modeling 

Three  noaa  laboratories — ^the  Geophysical  Fluid  Dynamics  Laboratory,  the 
National  Meteorological  Center,  and  the  Forecasting  Systems 

Laboratory— perform  cUmatological  modeling  on  supercomputers. 
However,  none  of  these  three  is  interested  in  using  ntf’s  supercomputers. 
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Potential  Civilian  Agency  Use  of  Excess 
NTF  Supercomputing  Resources 


The  Geophysical  Fluid  Dynamics  Laboratory  did  not  find  ntf’s  proposed 
price  for  leasing  time  on  its  Cray-2  attractive  since  the  laboratory  already 
has  access  to  a  newer,  more  cost-effective  supercomputer — ^the  Cray 
Y-MP.  The  National  Meteorological  Fluid  Dynamics  Laboratory  also  found 
the  hourly  rate  that  ntf  proposed  to  be  limiting  when  the  costs  of  software 
conversion  and  communications  were  included.  The  Forecasting  Systems 
Laboratory  already  has  access  to  supercomputers  and  does  not  have 
additional  funding  to  lease  time  elsewhere.  Furthermore,  laboratory 
officials  are  not  interested  in  the  Cray-2  because  it  is  an  old  technology. 
Instead,  they  said  they  would  be  interested  in  obtaining  access  to  newer 
generations  of  Cray  machines,  such  as  the  Y-MP  C90  or  the  massively 
parallel  T3D. 
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Major  Contributors  to  This  Report 


Accounting  and 
Information 
Management  Division, 
Washington,  D.C. 


John  A.  de  Ferrari,  Assistant  Director 
Elizabeth  L.  Johnston,  Evaluator-in-Charge 


Denver  Regional 


Jamelyn  Smith,  Senior  Information  Systems  Analyst 


Office 


(511293) 
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